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ABSTRACT 

A Study of the requisite processes for establishing 
diagnostic validity in reading was conducted in a seventh grade 
remedial reading classroom. The study's objectives were to (1) 
collect reliable diagnostic data on each student's performance in 
word recognition, oral reading, silent reading comprehension, and 
listening comprehension before and after the year's instruction; (2) 
docifflent the teacher's instructional practices; (3) link outcomes 
with instruction; and (4) return diagnostic and outcome information 
to the teacher. On the basis of this information, the teacher made 
instructional adjustronts for the next year's program in the areas of 
word recognition and oral reading. The diagnostic results for the 
following year showed improvement in student achievement in those 
areas. The results also indicated that an unobtrusive system for 
establishing diagnostic validity can be successfully introduced into 
the classroom. (Author/HTH) 
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Abstract ' 

Leamlag fro« experience to iaprove the teaching of reading depends on 
teachers havln« access to uaabU information about the effect of their prac- 
Uce on student tmtcoaws. Reliable diagnostic classification, documentation 
of Instruction, and t\e measurement of outcomes are needed to esUbliah a sys- 
tem for veriffing the links that connect diagnosis with Instruction and out- 
comes. A study of the requisite processes for establishing diagnostic 
validity in reading was conducted in a seventh-grade remedial -reading class- 
room. The study's objectives were (I) to collect reliable diagnostic da u on 
each student' a performance In word recognition, oral reading, silent reading 
comprehension, and listening comprehension before and after the year's in- 
struction; (2) to document the Uacher's instructional practices^ O) to link 
outcomes with instruction; and (4) to return diagnostic and outcome informa- 
tion to the teacher. On the basis of this Information, the teacher made 
instructional ^cislons for the next yeafs program In the areas of word 
recognition and oral reading. Outcome results the following year showed 
improvement in student achievement In those areas. The results indicate that 
an unobtrusive system for establishing diagnostic validity can be successfully 
introduced Into the classroom. t. 
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LEUVRNIHG FROM EXPERIENCE TO IMPROVE (HITCOMES 
IN REAOINCt A CASE STUDY ^ 

A AnMtte Weittslumk, Ruth Polln, attd Christian Wagner^ 

n 

How can teachers Uam s»r« effectively froa their own practice? 
Experience i»y Indeed be a veluable teacher, hut it has a serious lUltatlon: 
CUesrooa events happen too quickly for teachers to aggregate reliably the 
results of Instruction in order to profit fro« them. Isolation and over- 
reliance on personal experience further restrict the range of loforaatlon 
available to a teacher that can be used to evaluaee Instructional de<iislons 
snd actions in light o* student achievement outcomes . 

absence of specific information linking student otTtcoaes back to 
reliable diagnosis and instruction has been a major obstacle to teachers of 
reading who want to learn systematically from their own experience. Improving 
student outcomes In reading requires that the teacher receive information that 
reliably claaslf Iss student performance into appropriate illagnostlc cate- 
gories, because assurance of diagnostic reliability is central to evaluating 
one's instructional practices. If the diagnostic categorisations are unre- 
liable, it is Impossible to determine whether Instruction was Inadequate for 
the student's problem or whether student performance was Incorrectly diagnosed 
and good Instruction tias wasted on Lhe wrong problem. 

Unfortunately, diagnostic unreliability pervades the field of reading. 
Only recently has it become possible to remedy the situation and to provide 



^Thls paper watf presented at the American Educational Research. 
Association meetings held April I984^ln Mew Orleans, Louisiana. 

^Annette V^lnahank la an IRT teacher collalrtjrator and ci^irdlnator ofc the 
Ou&c^-ncs In Reading Project. Ruth Polio Is « research associate for that 
project. Christian Uagne,r, mho teaches In the College of Engineering at 
Oakland University, serves aa the project's consulunt. 
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teachers ulth reliable diagnostic inforMtioti. A series of studies on the 
diagnostic reliability of educational practitioners (Vinsonhaler, Ueinshank, 
Wagner, & Polln, I983j Ueinshank, 1982; Ueinsbank i Vlnspnhaler, 1983) showed 
that reading and learning disability specialists and classrooa teachers did 
not agree with theaselves or with one another in theit^ diagnostic judgnents 
about siattlat«d cases of children with reading problens. A second iteries of 
studies (Vftosotthaler, Ueinshank, Polin, & Uagner, 1983) showed tliat the 
reliability of diagnostic, decision Baking could be isproved dramatically 
through the use of a specific type of training progras. 

Two laportant producta of the training -study series were (I) a reading 
diagnostic teat battery based priaarily on perforaance in InsUnt word recog- 
nition, decoded word recognition, oral reading, silent reading c<MBprehen8ion, 
and listening coaprehension and (2) a conputer program for generating a reli- 
able diagnosis given the resulting student perforaance data. Thus it becaae 
possible to provl-^e teachers with reliable diagnostic Inforaation about their 
atudents' reading perforaance. 

This paper presents a case study of a Junior high school renedlal reading 
teacher who was provided with reliable diagnostic infornstlon about her stu- 
dents' reading perforaance and was thus able, for the first tiae, to learn 
ays teas tics I ly froa her own experience and aake changes for the better in her 
iostruction. 

A ase Study of Diagnostic Vslidtty 
The ability to train practitioners to be aore relisble and/ or have reli- 
able diagnoses provi«iad thea becaae the bssis for a series of ootcooe studies 
in the senior suthor's seventh -grade pull-out reaedlal reading prograa. (The 
senior suthor is a teacher cellsborator with the Institute for Siesearch on 



TcAChlng, condactlng rateareh half-tiae and teaching half-tiae in the public 
•choola.) The studies were designed to test the methods needed to aove from 
diagnostic reliaM^ity to diagnostic validity. Given reliable diagnostic 
information, what instructional plans do teschers carry out and with what 
effect on nfudents? 

# 

During the study's first year (Wagner, 1982), the diagnostic bsttery was 
administered to each student in the remedial -reading program as a pretest and 
a posttctc., using diagnoatic decision aids developed in the training series, 
the teaciu r diagnosed each student. Using the same data, the conputer program 
also ^aerated a diagnosis for each student. Agreement between teacher and 
computer was better than 90S. The remedial instruction that the teacher 

implemented with her students was documented by her dally notes end interview 

V 

and observational data collected by project sUff. At year's end, gain scores 
across the diagnostic categories were computed for atl students. 

The information on. student gains confirmed some of the teacher's subjec- 
tive evaluations and disconf irmed others. Much lower than anticipated gains 
In oral reading led her to alte?r instructional procedures for the next year's 
atudents. With <the help of an instructional aide, the teacher instituted two 
isajor changes ia f»Bt practice. First, she decided to abandon drill on comon 
word families; her expectation that the students would generalize the patterns 
to unfamiliar words encountered in text was not supported, by the p^rformaj)ce 
scores. Instead, »she would draw vocabulary words exclusively fr'.»m the mate- 
rials the studenU nad. Students would have to master these words to speed 
and accuracy criteria before reading text. Syllabication strategies and 
* soMOd-symbol association drill would be embedded within the vocabulary fiords. 
The tescher also decided that instead of conducting group oral reading f a 
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portion of each MSsion, sba ifouU require daily oral reading on on individual 
basis. 

loatmctioaol Coala 

Th« teachov said her ganaral instructional goals were iaproved word 
rocognitioB and»aaalysi8 skills within the context of extended, supervised 
orsl rcadins. She had tmo «aJor goals: <l) autoMte atkslysis and resognition 
skills in • fraaework of contextual reading and (2) give the students confi- 
dence in their ability to control the flow of print visually and orally. She 
aaid that direct instructlwi in c«Bprehension ran second to these two instruc- 
tional goals. In her judgnaat, instructional tiae constraints coupled with 
serioas deficits in word snalysis and fluency dicuted a primary esphasis on 
bringing students up to grade placeaent on those skills. Coaprebension probes 
^jj^re used inform I ly within the oral reading context. 

The Reading Prograa 

Five groups of four to six students caae to the reading rooo froa their 
respective clasarooos for a 25-einute session four days a week betneen October 
end Hay. for a tore I of 40 instructional hours. The students aoved through a 
fi«ed sequence of aaterials, entering at a level deterained by the resulU of 
their perforaance on the word recognition and oral reading subtests of the 
battery and generally aoving through the sequence st their own pace. Soae- 
tiaes. students were allowed to skip certain steps. The prograa functioned 
aore like a clinic than a classrooa. Stu dents read orally twice during each 
session. Lists of vocabulary words preceded various subsets of text, and 
those words had to be aaatared to speed and accuracy criteria before the stu- 
dent was allowed to begla oral reading. 
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Tb« teacher ttescrlbad Iwr program as staged. In Sugc I, students used a 

t 

lingMlatlsally based reading pfcograa. This type of program typically surts 
by preseotiog aiogle -syllable words »ade up of highly regular sound -sysbol 
correspondences and then proceeds slowly to sore irregular ones. Stuilents / 
were placed in reading Uxts appropriate to their levels of perforaance. No 
worksheets or isolated phonics drills were used. In Stage 2, e ssall nuaber 
of -syllabication heuristics were presented so thst the previously aastered . 
sound*synbol correspradences could be transferred to two- and three -syllable 
words. Content-based texts were used during this phase. Stage 3 was designed 
to fiove students toward fluency and autoaaticity with texts of increasing dif-" 
ficulty. A variety of basal texts at grade levels 6 and 7 were used for con- 
textual practice. 

The teacher recorded oral reading perforaance throughout the year. Every 
tiae a student read, the date, page(8) read, efapsed tiae, and words alscailed 
were written on a sheet devoted to the particular text the student- wss working 
with at that tiae. Text read at a very slow rate and/or with excessive ais- 
calLs wss repested and docuaented on a record sheet. so thst students could 
note changes in their own performance. After coapleting an oral reading 
session, the student continued reading silently slone until cslled back 5-10 
nlnutes later Co raad orally again. 

The procedures for the second year's study were tM (I) collect, using the 
diagnostic bettery, rellsble diagnostic data on each student before the year'j 
Instruction began and return the perforaance inforaatlon to the teacher-, 
(2) repeat diagnostic testing st the end of the year and return the perfor- 
aance inforaatlon "o the teacher; <3) detensine student outcoaes and return 
that Inforaatlon to the teacher; and (4) exaaine the links between Instruction 
and outcoaes. 



Collectia^ R«lUbU DUBPoattc DaU 

At the begittnlnt end end of the year, the diagnostic battery was 
adainiatcced iodlvidually by trained teaters to the 23 students in the reading 
progran, ali of whom began the progras reading two or nore ye'ara below grade 
placeaeat. The battery included Miasures of known reliability and provided 
'date on aeven variables of interest: (1) word . jcogni tion and (2) percent of 
words correctly identified by decoding (Slosson Oral Reading Teat), (3) oral 
reading oi paragraphs based on nunber of siscalls and (4) mean words .per 

■inute (Cray Oral Reading Paragraphs), (5) silent reading cotaprehension based 

< 

on percent of, passage recalled and (6) aiear. words per ainute (silent reading 
test of flie Durrell Analysis of Readl^ng Difficulty), and (7) listening cosipre- 
hension based on percent of passage recalled (listening conprehension test. of 
the DarrellAnalyiis of Reading Difficult/). 

f 

Rctmntnt Pretest Ittforaation to* the Teacher 

The teacher received pretest diagnostic information in two different 
forma: (I) noperically, as individual and nean group scores for each of the 
seven diagnostic catcgorfvs, and (2) graphically, as performance charts. The 

four charts .(word recognition and percent decoding, oral reading grade 

o 

equivalent vs. rate, silent reading grade equivalent vs. rate, and listening 
conprehension grade equivalent) allowvsd the teacher to see group configura- 
tions before and after Instruction and to ttfach changes for individual 
students. Table I shdus a pretest perforasnce chart for oral reading. 

In thia class, 87X of the students antered reading orally at three or 
■ore years be'low grade plaeetteat. by the end of the year, that figure had 
dropped to 57%, There was also a trend toward increased reading rate. On the 
individual level. Student 1014, for exasple, entered the progras reading at a 
third-grade level and exited reading at en ei^th-grade level. 

O 10 
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Table I 



Oral Rca41iigt Pretest PerfcnBence 
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Fpattait and Ontceaw loforontioo > 

The diagnostic bttt«ry wi9 ftgaln adbiolfltered to all atudenta after the 
year*t inatruetlon was over* Valuaa for each student (posttest score alnus 
pretest leore) for ee«li ot tKe diagnostic categories were calcyLated. Post- 
test and outcoae iaforaation was returned to the teacher in nuaerlc and 
graphic fora. The posttest and outcowe charts for oral reading are shown in 
Tables 2 and 3. 

OtttcoM reaults show the differential effects of the teacher's 
inatruction, at least aa asasured by the aiteinistered diagaoatle teats. The 
students stwwed strong gaina in*'oral reading accuracy with so«e signs of 
inereaaed rato. Their word recognition skills Uproved, and their use of 
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Table 2 

^OmI Readlogs Posttett Perfonaaace 
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Table 3 
Oral Readings Outeoaes 
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decoding «• a AtraUgy for »tord attack increased. Perfonaance on silent 
reading coapreliension paragraphs vaa not as encouraging. A trend toward in- 
creased silent reading rate suggests that the students aay well have been 
reading both -ore fluently and aore accurately given their oral reading and 
word recognition performance, hot these subskills did not appear to affect 
coaprehenaion of the test paragraphs. Listening coaprehension showed the 
least change. In general, the students seemed to find the subject matter of 
the paragraphs as obscure at the end of the year as at the beginning. 
To summarise, tbe outcome results were as follows: 

1. Word recognition . The mean gain was 1.3 years in grade 
level (range - .3-2.3); mean ftercent of words decoded 
rose from 6. 5X on the pretest to 16% on the posttest. 

2. Oral reading . The mean gain, as measured by number of 
isiscails, was two grade levels; mean words per minute 
remained essentially unchanged (84 words per minute on 
the pretest and 87 wrds per minute on the posttest). 

3. Silent readin£ comprehension . The mean gain Has half a 
grade level (range " 0-3.0}; mean words per minute in- 
creaaed 7.4% (100 words per minute on the pretest; IOC on 
the posttest). 

. 4. Listening comprehension . There was essentially no change 
in mean performance (grade level 5.0 on the pretest; 
grade level 5. 1 on tlw posttest). 

Linking Instructional Prafc'tices to Outcomes 

The overall outcome results confirmed the relative efficacy of the teach- 
«r's Instruction in word recognition and oral reading and Its relative lack of 
impact oo silent reading comprehension and listening comprehension. A more 
fine-grained examinati^ of the links that connect diagnosis with instruction 
and outcomes ia «ade possible by observing the effects of the instructional 
yrogrsm on students in any given pretest cell of a performance chart. If each 
cell does, in fact, represent a diagnostic category that correctl, separate* % 



13 



10 

like froo unlike fttudenU, then it should be possible to pinpoint differential 
effects of instruction across categories. 

OtttcoM results for both word recognition and oral reading for students 
nith identical pretest scores were analyaed* Results §how that the teacher 
was MSt effective with the sost disabled of the students in both word recog* 
nition and oral reading. In word recognition, the strongest gains were nade 
by the students who entered three or «ore years below grade placement. In 
oral reading, the strongest gains were made by students who entered four 
years belo. grade placeafent. Of the II students In this catego nine aade 
gains ranging froa oAi to four years. The effects of instruction on oral ^ 
reading rate are asbiguous. Strong gains in grade-level placement were as 
often associated with increases in rate as with decreases. 

' Analy^ng word, recognition performance on the dimension of percent 
decoding showed that all students in the 5% decoding category Increased their 
level of decoding Irrespective of grade-level placement. However, of the stu- 
denta in the 15% category, only those who entered three or more years below 
grade placement increased tho number of words arrived at via decoding. This 
appears to further confirm the efficacy of the program for ^e weakest read- 
ers. 

An analysis of the teacher's Instructional documentation for oral reading 
provided information, by student, on oral-reading rate, mmber of words read 
orally, number, of times (trials) a selection was reread, a list of all words 
that were miscalled, and attendance. Findings for this cUss include the 
following! 

1. Hean rate for oral reading during instruction was 63 + 7 
words per minute. This was slower than the rate for 
oral reading paragraphs both on the pretest (84 Hh 17) and 
the postteat (87 -¥ 12). 
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2. Total Man minber of uords read orally during the school year 
was approxlaataly 26,000. Students who nade the strongest 
gains i»er« belo» the eean; they read approximately 20,000 word«. 

3. Mean mmbar of trials was 1.5. Repeated trials (aore than 2) 
wera effective la reducing aisc^ais but .ineffective or counter- 
producti^-e in increasing rate. The teacher has learned that 
having students read a selection aore than twice is an exercise 
in diainiahing i^Jtumsi students would he better off reading 
another sa lection of coaparable difficulty. 

4. Across all 23 students, 2,478 Hpvia were aiscalled during oral 
reading. Of the list words aastered pi^ior to oral reading, 
only 15X wera snbsequently aissed, suggesting that list 
aaatery probably contributed to fluency. 

5. Mean attendance was 82%; aost of the students cane to m»st 
of the sessions. 

For tha first tlae in her teaching career, this teacher received specific 
ioforaation that enabled her to linl^^y to day instructional activities with 
student achieveaent in reading. On the basis of this Inforaation she aade 
seferal decisions about the next year'6 prograo: (1) continue using text- 
based vocabulary for decoding and syllabication practice, (2) limit repeated 
reading of a passage to two trials only, (3) elieinate ^se of the 
linguistically -based aaCerials except for the very weakest students (stronger 
students will enter the sequence at Stage 2), and (4) maintain comprehension 
Instruction at the same infonsal level. 

i 

Establishing Diagnostic Validity Within and Across^ Classrooms 
The results Indicate that it is polisible to introduce into a classroom 
setting a system for establishing diagnostic validity, that is, establishing 
the links that connect diagnosis with instruction and student achieveaent. 
The procedures sre non-intrusive and sensitive to the teacher's regular in- 
structional practices. Receiving inforaation about the results of instructioi 
aay or may not persuade a teacher to alter pvactlce, but It does provide a 
rational basis for decision asking. This teacher found that overall, her 
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iostruetloiMil prograa scrred her goals of.lBprovlng word recognition and oral 
reading skills. Sh« learned that ter program is more effectlTe for the lowest 
achieving entering aeventh graders then It Is for those somewhat less dls-- 
ablad. She has decided to alter her Instructional strategy for the sttonger 
entering students hut has not decided how (or whether) to expand her goals tp 
Include nore direct Instruction in coRprehenslon* 

The teacher can continue to learn ftom her am teaching practice by 
repeating the documentation of the diagnostic^ tree tsent-outcose cycle year 
after year. But tht. process Is slow and there la no CMparatlve Information 
available to her from other teachers who deal with similar students. How do 
students with similar entering characteristics l£are In classroom settings that 
sre similar to or different from hers? How do they fare with similar or 
different materials and strategies? 

Diagnostic validity for students who range across the whole spectrum of 
reading perfonwnce can be established over time only by practitioners willing 
to learn from their own practice and frms that of their colleaiguese Learning 
from experience to Improve practice In reading depends on teachers having 
access to reliable Infoimatlon about the cMitcoses of their own and their 
colleagues' Instructional practlcese In a study currently under way, the 
authors are extending the procedures described for establishing diagnostic 
iralidlty to 10 fifth-grade classrooms. A major goal of the research Is to 
document a full range of diagnostic classifications , associated instruction, 
and student achievement In reading. Such documentation would provide a bench- 
mark against which teachers and researchers could measure the consequences of 
instruction both within and acroas classroomb. The practitioners Involved In 
the slady will be in a poaition to learn from and share their o%m and their 
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colleague* • •npirlence while slmulUneouely contributing to the growth of 
deecriptlve, aod uUlmately prescriptive, data hase that connect, dlagnosl 
with inatructl-on and outcoaefi. 

s 
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